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1.  Purpose 


1.1  Background  and  Purpose 

Effective  aircrew  performance  is  critical  to  mission  success.  Crewstations  that  are  designed  to 
augment  the  cognitive  and  physical  abilities  of  aircrews  will  help  minimize  pilot  workload, 
enhance  situational  awareness  (SA),  enable  effective  crew  coordination,  and  contribute  to 
successful  mission  performance.  It  is  vital  that  crewstations  be  assessed  early  and  often  during 
development  to  ensure  optimal  design. 

The  U.S.  Army  Research  Laboratory,  Human  Research  and  Engineering  Directorate  (ARL 
HRED)  assesses  crewstation  design  for  new  and  upgraded  Army  Aviation  aircraft.  The 
assessments  are  conducted  to  identify  and  eliminate  human  factors  design  problems.  The 
methodology  used  to  assess  crewstation  design  includes:  anthropometric  modeling;  simulation 
and  operational  testing,  to  evaluate  pilot  workload,  SA,  crew  coordination,  and  pilot-crew  station 
interface  (PCI);  anthropometric  accommodation;  and  use  of  a  head  and  eye  tracker  to  assess 
visual  gaze  and  dwell  times.  This  methodology  has  been  used  by  ARL  HRED  to  help  develop 
all  modernized  Army  Aviation  systems,  including  the  AH-64D/E  Apache  Longbow,  UH-60M 
Blackhawk,  CH-47F  Chinook,  OH-58F  Kiowa,  and  UH-72A  Lakota.  This  report  provides  an 
overview  of  the  assessment  methodology  used  to  ensure  that  Army  Aviation  crewstations  are 
designed  to  help  pilots  perform  their  flight  and  mission  tasks,  and  summarizes  results  that  were 
used  to  drive  crewstation  design  changes. 


2.  Method 


2.1  Anthropometric  Modeling  and  Measurement 

ARL  HRED  developed  and  maintains  a  digital  model  library  of  Army  Aviation  aircraft, 
associated  equipment,  and  newer  aircraft  designs  that  are  in  the  conceptual  phases.  Human 
factors  analysts  use  the  digital  model  information  to  compare  Army  Aviation  aircraft  and 
equipment  design  to  current  human  factors  engineering  standards.  The  analysts  use  the  modeling 
results  to  assess  anthropometric  requirements,  improve  the  ergonomic  design  and  functionality  of 
the  systems,  and  reduce  analysis  timelines.  When  a  physical  prototype  or  first  article  hardware 
system  is  available  for  evaluation,  anthropometric  analyses  of  10-15  critical  human  dimension 
metrics — stature,  bideltoid  breadth,  chest  depth,  butt -knee  length,  interpupillary  breadth, 
functional  leg  length,  hand  length,  hand  breadth,  thumb  tip  reach,  and  sitting  eye  height 
(Donelson  and  Gordon,  1991) — are  used  to  ensure  that  the  participants  represent  a  broad  range 
of  the  intended  user  population  with  respect  to  human  dimensions.  For  example,  sitting  eye 
height  would  be  used  to  determine  if  a  small  female  (5th  percentile  of  the  target  population)  could 
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attain  the  appropriate  sitting  height  for  field  of  view,  and  still  manipulate  all  required  controls 
and  equipment  in  each  reach  zone  of  a  newly  designed  cockpit.  In  this  case,  human  figure 
modeling  (HFM)  and  physical  measurement  would  be  used  to  determine  whether  there  are  any 
early  limitations  with  respect  to  human  accommodation. 

2.2  Anthropometric  Modeling  Software 

As  part  of  the  anthropometric  modeling  process,  ARL  HRED  uses  the  Jack1  HFM  software  to 
assess  the  ergonomic  design  of  aircraft  systems.  Jack  is  an  interactive  tool  for  modeling, 
manipulating,  and  analyzing  human  and  other  3-dimensional  (3-D)  articulated  geometric  figures 
(Badler,  Phillips,  and  Webber,  1993).  The  software  also  contains  a  utility  for  importing 
anthropometric  data  that  can  be  used  to  build  and  size  the  human  figure  models.  This  allows  the 
human  factors  analyst  to  develop  the  models  to  represent  a  specific  user  population  for  whom  the 
equipment  is  targeted. 

Computer-based  graphical  human  figure  models  have  been  used  to  perform  ergonomic  analyses 
of  workplace  designs  since  the  late  1960s  (Das  and  Sengupta,  1995).  This  method  has  gained 
widespread  acceptance  over  the  past  two  decades,  as  designers  have  migrated  from  traditional 
paper  drafting  methods  to  the  use  of  computer-aided  design  (CAD)  software.  These  HFM 
programs  have  proven  to  be  an  effective  tool  for  evaluating  the  physical  interaction  between  the 
human  and  the  equipment.  Figure  1  shows  a  model  of  the  digitized  aviation  life  support 
equipment  worn  by  pilots  and  used  to  assess  crewstation  design,  along  with  a  small  female  pilot 
seated  in  a  UH-60M  crewstation  and  her  projected  line-of-sight. 


Figure  1 .  Model  of  aviation  life  support  equipment  and  female  pilot  line-of-sight. 

ARE  HRED  has  used  HFM  to  assess  anthropometric  requirements,  visual  obstructions,  physical 
reach,  flight  control  envelopes,  Air  Warrior  life  support  equipment  integration,  and 


1  •  • 

Mack  is  a  registered  trademark  of  Siemens. 
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pilot/equipment  space  restrictions.  We  authored  a  previous  report  describing  past  modeling 
efforts  and  results  (Hicks,  Durbin,  and  Kozycki,  2010). 

2.3  Simulation 

The  simulators  used  by  ARL  HRED  for  crewstation  design  assessments  are  engineering 
simulators.  The  engineering  simulators  represent  the  intended  production  design  and  provide  a 
platform  for  developing  and  assessing  crewstation  design,  evaluating  pilot  performance,  and 
assessing  crew  workload,  SA,  and  crew  coordination.  Ten  pilots  with  various  levels  of 
experience  (e.g.,  500-4000  flight  hours)  typically  participate  in  the  simulation  events.  This  wide 
range  of  experience  provides  ARL  HRED  researchers  a  broad  perspective  on  the  design  of  each 
crewstation.  Pilots  use  the  simulators  to  perform  tasks  and  fly  representative  missions  (e.g.,  zone 
reconnaissance,  call-for-fire,  troop  transport).  The  simulators  are  also  used  to  help  pilots  develop 
tactics,  techniques,  and  procedures  (TTP)  and  provide  limited  training  for  pilots  prior  to 
operational  testing  in  the  aircraft.  Results  of  the  assessments  are  provided  by  ARL  HRED  to  the 
aircraft  program  managers,  Training  and  Doctrine  Command  (TRADOC)  Capabilities  Managers 
(TCM),  Army  Test  and  Evaluation  Command  (ATEC),  Aviation  and  Missile  Research, 
Development  and  Engineering  Center  (AMRDEC),  and  defense  contractors. 

Simulators  previously  employed  by  ARL  HRED  include  the  OH-58F,  AH-64D  Apache 
Longbow  Risk  and  Cost  Reduction  Simulator  (RACRS);  UH-60M  Blackhawk  Helicopter 
Engineering  and  Analysis  Cockpit  (BHEAC)  and  Systems  Integration  Laboratory  (SIL) 
simulators;  CH-47F  Chinook  Helicopter  Engineering  and  Analysis  Cockpit  (CHEAC);  Armed 
Reconnaissance  Helicopter  (ARH)  simulator;  and  the  RAH-66  Comanche  Engineering 
Development  Simulator  (EDS)  and  Comanche  Portable  Cockpit  (CPC).  The  simulators 
contained  the  hardware  and  software  that  emulated  the  controls,  flight  characteristics,  and 
functionality  of  the  aircraft.  The  simulator  crewstations  replicated  the  corresponding  crewstation 
in  the  actual  aircraft,  allowing  each  pilot  to  perform  realistic  flight  and  mission  tasks.  The  OH- 
58F,  BHEAC,  CHEAC,  and  ARH  simulators  were  housed  in  the  Battlefield  Highly  Immersive 
Virtual  Environment  (BHIVE).  The  BHIVE  is  an  immersive  environment  in  which  the 
simulation  events  are  conducted,  that  provides  a  high  fidelity  out  the  window  display.  Table  1 
lists  the  aircraft,  associated  simulator,  virtual  environment,  and  assessment/test  for  which  the 
simulation  was  conducted. 
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Table  1.  Army  aircraft,  associated  simulator  and  assessment/test. 


Aircraft 

Simulator 

Assessment/Test 

OH-58F 

OH-58F  -  BHIVE 

Human  Factors  Engineering  (HFE)  #1,  2,  3 
Design  Assessment 

AH-64D 

RACRS 

Unmanned  Aircraft  System  (UAS)  Teaming 

ARH 

ARH  -  BHIVE 

Common  Aviation  Architecture  System 
(CAAS)  Assessment 

CH-47F 

CHEAC 

Common  Aviation  Architecture  System 
(CAAS)  Assessment 

RAH-66 

CPC,  EDS 

Force  Development  Test  and  Experimentation 
(FDT&E)  1 

UH-60M 

BHEAC,  SIL 

Early  User  Demonstration  (EUD) 

Limited  User  Test  (LUT) 

Limited  Early  User  Evaluation  (LEUE) 

As  examples,  figure  2  shows  the  OH-58F  crewstation  simulator  and  figure  3  shows  the  AH-64D 
Apache  Longbow  crewstation  simulator. 


Figure  2.  OH-58F  crewstation  simulator. 


Figure  3.  AH-64D  Apache  Longbow  crewstation  simulator. 
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2.4  Simulator  Sickness  Questionnaire 


During  the  simulations,  ARL  HRED  collects  and  analyzes  pilot  Simulator  Sickness 
Questionnaire  (SSQ)  ratings.  The  ratings  are  used  to  identify  whether  the  simulators  induced 
simulator  sickness  symptoms  (e.g.,  nausea,  headache),  if  the  symptoms  caused  significant 
discomfort  that  distracted  the  pilots  during  missions,  and  whether  the  symptoms  contributed  to 
an  increase  in  perceived  workload.  The  ratings  were  augmented  with  observations  by  ARL 
HRED  personnel  during  the  assessments,  pilot  feedback  during  post  mission  interviews,  and 
comparison  of  SSQ  ratings  with  ratings  from  other  helicopter  simulators  (table  2).  We  wrote  a 
previous  report  describing  past  simulator  sickness  data  collection  efforts  and  results  (Hicks  and 
Durbin,  2011). 

Table  2.  SSQ  scores  for  simulators. 


Simulator 

Nausea  Subscale 

Oculomotor 

Subscale 

Disorientation 

Subscale 

Total  Severity 
Score  (Mean) 

ARH  (BHIVE) 

18.02 

21.48 

9.28 

20.15 

OH-58F  (BHIVE) 

8.86 

21.32 

18.91 

19.23 

CH-47F  (CHE AC) 

12.52 

18.48 

10.15 

16.75 

RAH -66  (EDS) 

11.84 

14.98 

4.54 

13.25 

RAH -66  (CPC) 

6.73 

15.40 

4.32 

11.40 

UH-60M  -  LEUE  (BHIVE) 

6.36 

11.81 

3.09 

9.15 

AH-64D  -  Integrated  (UAS) 
(RACRS) 

9.01 

7.58 

4.64 

8.51 

UH-60M  -  EUD  (BHIVE) 

13.88 

6.89 

0 

8.5 

UH-60M  -  LUT  (SIL) 

6.36 

8.64 

2.71 

7.49 

AH-64D  -  Non-Integrated 
(UAS)  (RACRS) 

3.18 

5.05 

4.64 

4.98 

The  SSQ  (appendix  A)  was  developed  by  Kennedy,  Lane,  Berbaum,  &  Lilienthal  (1993)  and  is  a 
self-reported  checklist  of  16  symptoms.  The  16  symptoms  are  categorized  into  three  subscales. 
The  subscales  are  Oculomotor  (e.g.,  eyestrain,  difficulty  focusing,  blurred  vision),  Disorientation 
(e.g.,  dizziness,  vertigo),  and  Nausea  (e.g.,  nausea,  increased  salivation,  burping).  The  three 
subscales  are  combined  to  produce  a  Total  Severity  (TS)  score.  The  TS  score  is  an  indicator  of 
the  overall  discomfort  that  the  pilots  experienced  during  the  mission  (Johnson  2005). 

To  analyze  the  SSQ  data,  the  symptom  severity  scores  are  calculated.  The  first  step  is  to  sum  the 
values  for  each  symptom  (e.g.,  eyestrain,  nausea).  The  values  are  coded  by  a  specific  number 
corresponding  to  symptom  severity.  A  value  of  0  equals  “no  symptom”,  a  value  of  1 
corresponds  to  “slight”,  a  value  of  2  is  “moderate”,  and  a  value  of  3  equals  “severe”.  Each 
symptom  severity  subscale  score  is  calculated  by  summing  the  values  of  each  subscale  and  then 
multiplying  each  individual  sum  by  a  conversion  factor.  The  TS  score  is  calculated  by  summing 
each  subscale  and  multiplying  by  a  total  severity  factor.  A  higher  score  indicates  more  severe 
symptoms  and  an  increased  likelihood  of  simulator  induced  sickness.  Table  3  categorizes  the  TS 
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scores  as  proposed  by  Kennedy  (2002).  Table  2  gives  the  SSQ  scores  for  the  engineering 
simulators  used  by  ARL  HRED. 

Table  3.  Categorization  of  SSQ  Total  Scores. 


SSQ  Total  Score 

Categorization 

0 

No  symptoms 

<5 

Negligible  symptoms 

5-10 

Minimal  symptoms 

10-15 

Significant  symptoms 

15-20 

Symptoms  are  a  concern 

>20 

A  problem  simulator 

2.5  Head  and  Eye  Tracker 

To  help  assess  crewstation  design,  pilots  wear  a  head  and  eye  tracker  to  record  visual  gaze  and 
dwell  times  during  missions  conducted  in  the  simulators.  Recording  visual  gaze  and  dwell  times 
can  help  identify  improvements  that  need  to  be  made  to  crewstation  design.  For  example,  if 
pilots  spend  an  excessive  amount  of  time  viewing  the  crewstation  displays,  this  can  indicate  that 
the  displays  contain  information  that  requires  too  many  steps  (e.g.,  button  pushes,  interpretation) 
to  retrieve.  The  data  are  augmented  with  observations  by  ARL  HRED  personnel  during  the 
assessments,  pilot  feedback  during  post-mission  interviews,  and  comparisons  of  eye  tracker  data 
with  findings  from  other  helicopter  simulators.  Figure  4  shows  the  eye  tracker  mounted  onto  a 
pilot’s  helmet.  Table  4  shows  outside/inside  cockpit  visual  gaze  times  for  the  AH-64D,  OH-58F 
and  ARH,  and  UH-60M  simulators  during  visual  flight  rules  (VFR)  conditions.  We  authored  a 
previous  report  describing  head  and  eye  tracker  data  collection  efforts  and  results  (Hicks,  lessee, 
and  Durbin,  2012). 


Figure  4.  Mounted  eye  tracker. 
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Table  4.  Simulator  comparison  data. 


Simulator  (Attack/Recon) 

Seat 

Outside  Cockpit 

Inside  Cockpit 

AH-64D  -  Integrated  UAS 
(RACRS) 

Co-Pilot 

6% 

94% 

Pilot 

75% 

25% 

AH-64D  -  Non-Integrated 
UAS  (RACRS) 

Co-Pilot 

3% 

97% 

Pilot 

75% 

25% 

AH-64D  -  non-U  AS 
(RACRS) 

Co-Pilot 

3% 

97% 

Pilot 

75% 

25% 

ARH  (BHIVE) 

Co-Pilot 

7% 

93% 

Pilot 

61% 

39% 

OH-58F  (BHIVE) 

Co-Pilot 

7% 

93% 

Pilot 

63% 

37% 

Simulator  (Cargo/Lift) 

Seat 

Outside  Cockpit 

Inside  Cockpit 

UH-60M  -  EUD  (BHEAC) 

Co-Pilot 

___ 

___ 

Pilot 

72% 

28% 

UH-60M  -  LEUE 
(BHEAC) 

Co-Pilot 

26% 

74% 

Pilot 

61% 

39% 

UH-60M  -  LUT  (SIL) 

Co-Pilot 

28% 

72% 

Pilot 

86% 

14% 

2.6  Assessment  of  Pilot  Workload,  Situation  Awareness,  and  Crewstation  Design 

ARL  HRED  uses  a  battery  of  rating  scales  and  techniques  to  assess  pilot  workload  and  SA 
during  missions.  A  common  definition  of  pilot  workload  is  “the  integrated  mental  and  physical 
effort  required  to  satisfy  the  perceived  demands  of  a  specified  flight  task”  (Roscoe,  1985). 
Assessing  pilot  workload  is  important  because  mission  accomplishment  is  related  to  the  mental 
and  physical  ability  of  the  crew  to  effectively  perform  their  flight  and  mission  tasks.  If  one  or 
both  pilots  experience  excessively  high  workload  while  performing  flight  and  mission  tasks,  the 
tasks  may  be  performed  ineffectively  or  even  abandoned.  In  order  to  assess  whether  the  pilots  are 
task-overloaded  during  the  missions,  the  level  of  workload  for  each  pilot  must  be  evaluated. 

SA  can  be  defined  as  the  pilot’s  mental  model  of  the  current  state  of  the  flight  and  mission 
environment.  A  more  formal  definition  is,  “the  perception  of  the  elements  in  the  environment 
within  a  volume  of  time  and  space,  the  comprehension  of  their  meaning,  and  the  projection  of 
their  status  in  the  near  future”  (Endsley,  1988).  It  is  important  to  assess  SA  because  of  its 
potential  to  directly  impact  pilot  and  system  performance.  Good  SA  should  increase  the 
probability  of  good  decision  making  and  performance  by  aircrews  when  conducting  flight  and 
mission  tasks. 

2.6.1  Bedford  Workload  Rating  Scale 

The  Bedford  Workload  Rating  Scale  (BWRS)  has  been  used  extensively  by  the  military,  civil, 
and  commercial  aviation  communities  for  pilot  workload  estimation  (Roscoe  and  Ellis,  1990).  It 
requires  pilots  to  rate  the  level  of  workload  associated  with  a  task  based  on  the  amount  of  spare 
capacity  they  feel  they  have  to  perform  additional  tasks.  Spare  workload  capacity  is  an  important 
commodity  for  pilots  because  they  are  often  required  to  perform  several  tasks  concurrently.  They 
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perform  navigation  tasks,  communicate  via  multiple  radios,  monitor  aircraft  systems,  and  assist 
the  pilot  on  the  controls  with  flight  tasks  (e.g.,  maintain  airspace  surveillance),  all  within  the 
same  time  interval.  Mission  performance  is  reduced  if  pilots  are  task-saturated  and  have  little  or 
no  spare  capacity  to  perform  other  tasks.  The  pilots  complete  the  BWRS  (appendix  B) 
immediately  after  each  mission  to  rate  the  level  of  workload  that  they  experience  when 
performing  flight  and  mission  tasks.  The  rated  tasks  are  selected  because  they  are  estimated  to 
have  the  most  impact  on  aircrew  workload  during  the  missions.  The  ratings  are  compared 
against  the  workload  ratings  requirements  for  the  aircraft  (as  applicable)  to  determine  if  the 
crewstation  design  is  imposing  excessive  workload  on  the  pilots.  Table  5  shows  a  summary  of 
overall  workload  averages  collected  during  a  sample  of  simulations  and  operational  tests.  The 
ratings  indicate  that  the  pilots  typically  experienced  moderate  overall  workload. 

Table  5.  Overall  workload  averages. 


Bedford  Workload  Ratings  -  Overall  Workload  Averages 

System/Test 

Co-Pilot 

Pilot 

AH-64D  -  Integrated  fUAS) 

2.60 

2.90 

AH-64D  -  Non-Integrated  (UAS) 

3.30 

2.60 

RAH-66  -  FDTE  1 

3.08 

2.90 

ARH  -  CAAS 

3.71 

3.94 

UH-60M  -  LEUE 

3.33 

2.98 

UH-60M  -  LUT 

2.80 

2.58 

CH-47F  -  CAAS 

2.66 

2.70 

OH-58F  -  HFE  #2 

3.17 

3.00 

2.6.2  Situation  Awareness  Rating  Technique 

The  Situation  Awareness  Rating  Technique  (SART)  (appendix  C)  is  a  multi-dimensional  rating 
scale  for  pilots  to  report  their  perceived  SA.  The  SART  was  developed  as  an  evaluation  tool  for 
the  design  of  aircrew  systems  (Taylor,  1989)  and  assesses  three  components  of  SA: 
understanding,  supply,  and  demand.  Taylor  proposed  that  SA  is  dependent  on  the  pilot’s 
Understanding  (U)  (e.g.,  quality  of  information  they  receive),  and  the  difference  between  the 
Demand  (D)  on  the  pilot’s  resources  (e.g.,  complexity  of  mission)  and  the  pilot’s  Supply  (S) 
(e.g.,  ability  to  concentrate).  When  D  exceeds  S,  there  is  a  negative  effect  on  U  and  an  overall 
reduction  of  SA.  The  formula  SA  =  U  -  D  -  S)  is  used  to  derive  the  overall  SART  score.  The 
SART  is  one  of  the  most  thoroughly  tested  rating  scales  for  estimating  SA  (Endsley,  2000).  The 
pilots  complete  the  SART  immediately  after  each  mission  to  rate  the  level  of  SA  that  they 
perceived  while  performing  the  mission.  Additionally,  pilots  rate  their  perceived  level  (high- 
low)  of  SA  of  battlefield  elements  (e.g.,  location  of  enemy  units  or  other  aircraft).  These  data 
provide  ARL  researchers  information  on  how  well  the  pilots  perceive  the  simulation 
environment  and  potential  threats.  The  battlefield  elements  situation  awareness  questionnaire  is 
completed  in  conjunction  with  the  SART  questionnaire  after  each  mission.  Table  6  shows  a 
summary  of  overall  SART  score  averages  collected  during  a  sample  of  simulations  and 
operational  tests.  The  data  indicate  that  the  pilots  typically  experienced  moderate  levels  of  SA. 
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Table  6.  Overall  SART  averages. 


Situation  Awareness  Rating  Technique  -  Overall  Averages 

System/Test 

Co-Pilot 

Pilot 

AH-64D  -  Integrated  (UAS) 

18.40 

23.20 

AH-64D  -  Non-Integrated  (UAS) 

19.00 

21.30 

RAH -66  -  FDTE 

21.86 

22.40 

ARH  -  CAAS 

17.67 

17.22 

UH-60M  -  LEUE 

26.42 

25.25 

UH-60M  -  LUT 

28.28 

28.22 

CH-47F  -  CAAS 

23.83 

20.13 

2.6.3  Pilot-crewstation  Interface  Evaluation 

PCI  evaluations  are  used  to  examine  the  interaction  between  the  pilots  and  the  crewstation 
interface.  The  PCI  impacts  crew  workload  and  SA  during  a  mission.  A  PCI  that  is  designed  to 
augment  the  cognitive  and  physical  abilities  of  crews  will  minimize  workload,  enhance  SA,  and 
contribute  to  successful  mission  performance.  To  assess  the  PCI,  the  pilots  report  any  problems 
that  contributed  to  high  workload  and  low  SA  at  the  end  of  each  mission.  They  also  complete  a 
lengthy  questionnaire  (appendix  D)  at  the  end  of  their  final  mission.  The  questionnaire  addresses 
usability  characteristics  of  the  PCI  (e.g.,  software  interface,  control  reach,  and  button  presses). 

2.6.4  Subject  Matter  Experts 

Subject  matter  experts  (SME)  observe  the  missions  independently  to  rate  pilot  workload  and  SA, 
mission  success,  and  levels  of  crew  coordination  (appendix  E)  that  they  observe  during  the 
missions.  An  SME  is  typically  an  experienced  pilot  that  has  in-depth  knowledge  of  the  aircraft 
and  crewstation  being  assessed.  The  ratings  provided  by  the  SME  are  compared  to  the 
corresponding  test  pilot  ratings  to  identify  any  significant  anomalies  in  perceived  levels  of 
workload  or  SA  while  interacting  with  the  crewstation. 

2.6.5  Pilot  Interviews 

Pilots  are  formally  interviewed  about  their  performance  during  after-action  reviews  (AAR), 
where  the  mission  events  and  goal  outcomes  are  discussed.  Pilots  are  also  interviewed  by  ARL 
HRED  researchers  informally  throughout  the  test  process  to  gain  insights  into  procedures  and  to 
capture  any  additional  comments  or  perceptions  of  the  test  process  and  general  crewstation 
design.  Additionally,  pilots  complete  forms  providing  recommendations  for  improvements  to 
the  crewstation;  their  recommendations  are  addressed  in  future  design  iterations. 

2.6.6  Data  Analysis 

Pilot  responses  to  the  BWRS,  SART,  SSQ,  and  PCI  questionnaires  are  typically  analyzed  with 
descriptive  statistics  to  examine  means  and  percentages.  Further  analysis  is  conducted  using 
non-parametric  statistical  tests,  such  as  the  Wilcoxon  Signed  Ranks  Test  (WSRT),  to  compare 
pilot  ratings  between  seating  positions  (e.g.,  left  vs.  right)  and  aircraft  models  (e.g.,  Block  II  vs. 
Block  III).  The  WSRT  is  used  to  calculate  probability  values  for  data  comparisons  and  statistical 
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significance.  Eye  tracker  data  is  usually  summarized  into  areas  of  interest  (AOI)  segments  to 
determine  the  amount  of  heads-down  time  pilots  have  while  operating  the  system.  Finally,  SME 
and  pilot  interview  feedback  are  analyzed  to  provide  additional  information  about  trends  or 
anomalies. 

2.6.7  Operational  Testing 

In  the  final  stages  of  crewstation  development,  operational  tests  are  conducted  to  verify  design 
requirements  and  ensure  the  crewstation  is  ready  for  fielding.  ARL  HRED  participates  in 
operational  tests  and  typically  collects  the  same  data  as  was  collected  during  the  simulations. 
This  provides  a  historical  assessment  of  pilot  performance  and  crewstation  design.  Results  from 
the  operational  test  are  compared  to  the  simulations  to  ensure  improvements  have  been  made  to 
the  crewstation  design  and  to  identify  new  issues. 


3.  Summary 


The  methods  that  ARL  HRED  uses  to  assess  the  human  factors  characteristics  of  Army  Aviation 
helicopter  crewstations  have  been  successful  in  identifying  and  eliminating  human  factors  design 
problems.  To  date,  over  300  crewstation  design  issues  have  been  identified  and  resolved  for 
Army  Aviation  aircraft.  Workload,  PCI,  and  SA  data  collected  during  testing  have  been  used  to 
improve  the  crewstation  interface.  Examples  include  software  improvements  to  crewstation 
displays,  such  as  enhanced  functionality  and  presentation  of  display  pages  to  pilots,  improved 
color-coding  of  battlefield  graphics,  reduced  number  of  button  presses  to  display  information, 
enhanced  readability  of  display  map  pages,  and  improved  presentation  of  aircraft  operational 
limits  (figure  5).  Anthropometric  measurements,  eye  tracker  data,  and  human  figure  modeling 
have  resulted  in  crewstation  hardware  improvements  that  include  modifications  to  crewstation 
seats,  consoles,  and  glareshields.  These  modifications  are  designed  to  improve  visual  access  and 
physical  reach  to  displays  and  controls,  provide  improved  functionality  of  flight  helmets  and  helmet- 
mounted  displays,  and  optimize  crewstation  switch  location  and  function. 

In  summary,  the  benefits  to  using  the  crewstation  assessment  method  are  (a)  iterative  crewstation 
assessments  drive  continuous  incremental  improvements,  (b)  improvements  are  identified  in  near 
real-time  which  aids  rapid  modification,  (c)  identifies  crewstation  design  that  needs  further 
improvement,  (d)  issues  documented  for  one  aircraft  often  apply  to  new  or  updated  aircraft — 
helps  with  early  identification  of  issues  for  new  and  updated  aircraft,  and  (e)  results  feed  the 
assessments  used  by  acquisition  officials  to  determine  whether  to  manufacture  and  field  Army 
Aviation  aircraft. 

ARL  HRED  will  continue  to  use  and  improve  the  crewstation  assessment  methodology  to  meet 
the  demands  of  the  next-generation  aircraft  for  the  Army. 
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Added  capability  to  view  additional 
information  on  map  display 
( e  g  .  operati  onal  1 1  m  its )  to  increas  e 
pilot  situational  awareness 


Changed  bezel  button  logic  for 
map  and  sensor  display  modes 
to  enhance  usability 


Enhanced  color-coding  of  missile  symbology, 
radio  frequencies,  and  operational  limits  to 
increase  discrimination  and  readability 


Q]  ooooo 


ichoo  fp 


a 


Figure  5.  Software  improvements  to  crewstation  displays. 
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Appendix  A.  Simulator  Sickness  Questionnaire 


SSQ  Questionnaire 


Please  indicate  the  severity  of  symptoms  that  apply  to  you  right  now  by  circling  the  appropriate  word. 


Symptom 

0 

1 

2 

3 

a.  General  discomfort 

None 

Slight 

Moderate 

Severe 

b.  Fatigue 

None 

Slight 

Moderate 

Severe 

c.  Headache 

None 

Slight 

Moderate 

Severe 

d.  Eyestrain 

None 

Slight 

Moderate 

Severe 

e.  Difficulty  focusing 

None 

Slight 

Moderate 

Severe 

f.  Increased  salivation 

None 

Slight 

Moderate 

Severe 

g.  Sweating 

None 

Slight 

Moderate 

Severe 

h.  Nausea 

None 

Slight 

Moderate 

Severe 

i.  Difficulty  concentrating 

None 

Slight 

Moderate 

Severe 

j .  Fullness  of  head 

None 

Slight 

Moderate 

Severe 

k.  Blurred  vision 

None 

Slight 

Moderate 

Severe 

1.  Dizzy  (eyes  open) 

None 

Slight 

Moderate 

Severe 

m.  Dizzy  (eyes  closed) 

None 

Slight 

Moderate 

Severe 

* 

n.  Vertigo 

None 

Slight 

Moderate 

Severe 

** 

o.  Stomach  awareness 

None 

Slight 

Moderate 

Severe 

p.  Buiping 

None 

Slight 

Moderate 

Severe 

Vertigo  is  a  loss  of  orientation  with  respect  to  vertical  upright. 
Stomach  awareness  is  a  feeling  of  discomfort  just  short  of  nausea. 
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Workload 


Reduced  spare  capacity.  Additional 
tasks  cannot  be  given  the  desired 
amount  of  attention 


Little  spare  capacity:  level  of  effort 
allows  little  attention  to  additional 
tasks 


Very  little  spare  capacity,  but 
maintenance  of  effort  in  the  primary 
tasks  not  in  question 


Very  high  workload  with  almost  no 
spare  capacity.  Difficulty  in 
maintaining  level  of  effort 


Extremely  high  workload.  No  spare 
capacity.  Serious  doubts  as  to  ability 
to  maintain  level  of  effort 
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Pilot  Workload 


Rate  the  workload  for  the  Flight  and  Mission  Tasks  you  performed  during  the  mission  that  you 
just  completed.  Use  the  scale  provided  on  the  last  page  of  this  questionnaire.  Place  the 
workload  rating  in  the  blank  next  to  each  Flight  and  Mission  Task.  If  you  did  not  perform  a  task 
during  the  mission  that  you  just  completed,  place  an  X  in  the  non-applicable  (N/A)  column. 


Flight  and  Mission  Tasks 


Workload  Rating 


Maintain  Airspace  Surveillance 


Perform  Hover  Power  Check 


Perform  Hover  Out-Of-Ground-Effect  (OGE)  Check 


Perform  Radio  Communication  Procedures 


Perform  Hovering  Flight 


Perform  Visual  Meteorological  Conditions  (VMC)  Takeoff 


Navigate  by  Pilotage  and  Dead  Reckoning 


Perform  Electronically  Aided  Navigation 


Perform  Fuel  Management  Procedures 


Perform  VMC  Flight  Maneuvers 


Perform  VMC  Approach 


Level  of  Interoperability  (LOI)  2  with  UAS 


Perform  A  Running  Landing 


Respond  to  Emergencies 


Respond  to  Engine  Failure  in  Cruise  Flight 


Perform  Nose  Mounted  Sensor  (NMS)  Operations 


Perform  Digital  Communications 


Negotiate  Wire  Obstacles 


Perform  Instrument  Takeoff 


Perform  Non  Precision  Approach  (GCA) 


Perform  Precision  Approach  (GCA) 


Perform  Emergency  GPS  Recovery  Procedure 


Perform  an  Autorotation 
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Perform  Unusual  Attitude  Recovery 


Operate  ASE/transponder 


Respond  to  IMC  Conditions 


Perform  Refueling  /  Rearming  Operations 


Perform  Electronic  Countermeasures  /  Electronic  Counter- 
Countermeasures 


Transmit  Tactical  Reports 


Perform  Terrain  Flight  Takeoff 


Perform  Terrain  Flight 


Perform  Terrain  Flight  Approach 


Perform  Masking  and  Unmasking 


Perform  Terrain  Flight  Deceleration 


Perform  Actions  on  Contact 


Perform  Weapons  Initialization  Procedures 


Perform  Firing  Techniques 


Engage  Target  with  .50  Cal 


Engage  Target  with  Hellfire 


Engage  Target  with  Rockets 


Perform  Aerial  Observation 


Perform  Target  Handover 


Aerial  Observation 


Call  for  Indirect  Fire 


Perform  Multi -Aircraft  Operations 


Perform  Combat  Maneuvering  Flight 


Perform  Close  Combat  Attack 


Perform  Combat  Position  Operations 


Call  for  Tactical  Air  Strike 


Zone  Reconnaissance 


Route  Reconnaissance 


Area  Reconnaissance 


Aerial  Surveillance 


Overall  Workload  for  the  Mission 
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If  you  gave  a  workload  rating  of  ‘5’  or  higher  for  any  task,  explain  why  the  workload  was  high 
for  the  task. 
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Intentionally  Left  Blank. 
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Situation  Awareness 


Situation  Awareness  is  defined  as  “timely  knowledge  of  what  is  happening  as  you  perform  your 
right  or  left  seat  tasks  during  the  mission.” 


Situation  Awareness  Rating  Technique  (SART) 

DEMAND 

Instability  of  Situation 

Likeliness  of  situation  to  change  suddenly 

Variability  of  Situation 

Number  of  variables  which  require  your  attention 

Complexity  of  Situation 

Degree  of  complication  (number  of  closely  connected  parts)  of  the 
situation 

SUPPLY 

Arousal 

Degree  to  which  you  are  ready  for  activity 

Spare  Mental  Capacity 

Amount  of  mental  ability  available  to  apply  to  new  tasks 

Concentration 

Degree  to  which  your  thoughts  are  brought  to  bear  on  the  situation 

Division  of  Attention 

Amount  of  division  of  your  attention  in  the  situation 

UNDERSTANDING 

Information  Quantity 

Amount  of  knowledge  received  and  understood 

Information  Quality 

Degree  of  goodness  or  value  of  knowledge  communicated 

Familiarity 

Degree  of  acquaintance  with  the  situation 
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Rate  the  level  of  each  component  of  situation  awareness  that  you  had  when  you  performed 
‘flying  pilot’  tasks  in  the  right  seat  -or-  ‘non-flying’  pilot  tasks  in  the  left  seat  during  the 
mission  that  you  just  completed.  Circle  the  appropriate  number  for  each  component  of  situation 
awareness  (e.g.,  complexity  of  situation). 


Instability  of  situation: 
Variability  of  situation: 
Complexity  of  situation: 


DEMAND 

Low  1 - 2- . 3 - 4 - 5 - 6 - 7  High 

Low  1 - 2 - 3 - 4- . —5 - 6 - 7  High 

Low  1 - 2 - 3— . —4 - 5 - 6 - 7  High 


Arousal: 

Low 

Spare  mental  capacity: 

Low 

Concentration: 

Low 

Division  of  attention: 

Low 

SUPPLY 

1 - 2 - 3 . —4 - 5 - 6— . 7  High 

1 - 2 - 3 - 4- . -5 - 6- . —7  High 

1 - 2 - 3 - 4 - 5 - 6 - 7  High 

1 - 2 - 3— . -4 - 5 - 6 - 7  High 


UNDERSTANDING 

Information  quantity:  Low  1 - 2 - 3 - 4 - 5 - 6- . -7  High 

Information  quality:  Low  1 - 2- . 3 . — 4 - 5 - 6- . -7  High 

Lamiliarity:  Low  1 . 2 - 3— . —4 - 5 - 6 - 7  High 

Battlefield  Elements 

Rate  the  level  of  situation  awareness  you  had  for  each  battlefield  element  during  the  mission? 
(Place  an  X  in  the  appropriate  column  for  each  battlefield  element). 


25 


Battlefield 

Elements 

Very  High 
Level  of 
Situation 
Awareness 

Fairly  High 
Level  of 
Situation 
Awareness 

Borderline 

Fairly  Low 
Level  of 
Situation 
Awareness 

Very  Low 
Level  of 
Situation 
Awareness 

Location  of 
Enemy  Units 


Location  of 
Friendly  Units 

Location  of 
Non- 

Combatants 
(e.g..  Civilians) 

Location  of  My 
Aircraft  During 
Missions 

Location  of 
Other  Aircraft 
In  My  Flight 

Location  of 
Cultural 
Features  (e.g., 
bridges) 

Route 

Information 
(ACPs,  BPs, 
EAs,  RPs,  etc.) 

Status  of  My 
Aircraft  Systems 
(e.g.,  fuel 
consumption) 


Describe  any  instances  when  you  had  low  situation  awareness  during  the  mission: 
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Appendix  D.  PCI  Questionnaire 
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PV1.  The  following  table  lists  the  components  of  a  CAAS  crewstation.  For  each  component, 
indicate  whether  or  not  you  experienced  a  problem  using  the  component  in  a  quick  and  efficient 
manner  during  the  mission  you  just  completed.  Check  ‘Yes’  if  you  experienced  one  or  more 
problems.  Check  ‘No’  if  you  did  not  experience  any  problems.  Check  ‘Not  Used’  if  you  did  not  use 
the  component  during  the  mission  you  just  completed. 

•  Multifunction  Displays  (MFD) 

o  Vertical  Situation  Display  (VSD)  Yes _  No _  Not  Used _ 

o  VSD  Hover  (VSDH)  Yes _  No _  Not  Used _ 

o  Horizontal  Situation  Display  (HSD)Yes _  No _  Not  Used _ 

o  HSD  Hover  (HSDH)  Yes _  No _  Not  Used _ 

o  EOS  Yes _  No _  Not  Used _ 

o  Digital  Map  Display  (DMS)  Yes _  No _  Not  Used _ 


o  Warning,  Caution,  Advisory 

Display  (WCA)  Yes _  No _  Not  Used. 


o  Engine  Instrument  Caution 
Advisory  System  (EICAS) 

Yes 

No 

Not  Used _ 

•  Control  Display  Unit  (CDU) 

o  Initializing  CDU 

Yes 

No 

Not  Used _ 

o  Managing  GPS  /  Flight  Plan 

Yes 

No 

Not  Used _ 

o  Managing  COM,  NAV,  IFF  (CNI) 

Yes 

No 

Not  Used _ 

If  you  answered  “Yes”  to  any  of  the  questions,  describe  a)  the  problems  you  experienced,  b)  how 

much  the  problems  degraded  your  performance,  and  c)  any  recommendation  you  have  for  improving 
the  design  of  the  various  functional  components. 


PV3.  Please  answer  the  following  questions  regarding  the  Multifunction  Control  Unit  (MFCU): 

PV3-1.  Did  the  functionality  of  the  directional  control  and  switches  on  the  MFCU  perform  the 
actions  you  expected? 

Yes _  No _  Not  Used _ 
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PV3-2.  Was  the  sensitivity  of  the  directional  control  appropriate? 


Yes _  No _  Not  Used _ 

PV3-3.  Did  you  experience  abnormal  hand  discomfort  while  using  the  MFCU? 

Yes _  No _  Not  Used _ 

PV3-4.  Did  you  have  adequate  space  in  the  cockpit  to  use  the  MFCU? 

Yes _  No _  Not  Used _ 

If  you  experienced  any  problems  with  the  MFCU,  please  describe  the  problems  in  as  much  detail  as 
you  can  and  make  recommendations  to  correct  the  problems. 


PV4.  Did  you  have  difficulty  using  any  of  the  switches  on  the  collective  or  cyclic  grips? 


Collective  Grip 

Yes 

No  _ 

Cyclic  Grip 

Yes 

No  _ 

If  you  answered  “Yes”  for  either  flight  control,  please  list  which  flight  control  and  switch(es),  and 
the  problems  you  experienced  (e.g.,  confused  two  switches  due  to  similar  shape,  switch  too  hard  to 
reach). 


PV5.  Was  there  any  symbology  depicted  on  the  following  displays/pages  that  was  difficult  to 
quickly  and  easily  understand,  cluttered,  or  otherwise  difficult  to  use? 


Vertical  Situation  Display  (VSD) 

Yes 

No  _ 

VSD  Hover  (VSDH) 

Yes 

No  _ 

Horizontal  Situation  Display  (HSD) 

Yes 

No  _ 

HSD  Hover  (HSDH) 

Yes 

No  _ 

EICAS 

Yes 

No  _ 

Digital  Map  System  (DMS) 

Yes 

No  _ 
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If  you  answered  “Yes”  to  any  of  the  questions,  please  describe  a)  the  display/page,  b)  the  symbology 
that  was  difficult  to  understand,  c)  how  the  symbology  may  have  degraded  your  performance,  and  d) 
any  recommendations  you  have  for  improving  the  design  of  the  various  functional  components. 


PV6.  How  would  you  rate  your  ability  to  detect  the  following  occurrences  based  on  the 
characteristics  of  the  flight  displays? 


Caution  /  Advisory  (MFD) 

1  2  3  4  5 


Very  Somewhat  Borderline  Somewhat  Very 

Easy  Easy  Difficult  Difficult 


Warning  (Master  Warning  Panel) 

1  2  3  4  5 


Very  Somewhat  Borderline  Somewhat  Very 

Easy  Easy  Difficult  Difficult 

Entry  into  Operational  Limits 

1  2  3  4  5 


Very  Somewhat  Borderline  Somewhat  Very 

Easy  Easy  Difficult  Difficult 


Low  Luel  (MED) 

1  2  3  4  5 


Very  Somewhat  Borderline  Somewhat  Very 

Easy  Easy  Difficult  Difficult 


If  you  answered  “Somewhat  Difficult”,  or  “Very  Difficult”,  please  indicate  which  annunciation  you 
had  difficulty  detecting,  why  you  may  have  had  difficulty  detecting  it,  and  any  recommendations  to 
make  the  annunciation  more  easily  detectable. 
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PV7.  Based  on  the  missions  you’ve  conducted  this  week,  what  are  the  top  enhancements  that 
should  be  made  to  the  crew  station  to  improve  pilot  performance? 
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Intentionally  Left  Blank. 
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Workload 


Insufficient  spare  capacity  for  easy 
attention  to  additional  tasks 


Reduced  spare  capacity.  Additional 
tasks  cannot  be  given  the  desired 
amount  of  attention 


Little  spare  capacity:  level  of  effort 
allows  little  attention  to  additional 
tasks 


Was  workload  tolerable 
for  the  mission  tasks? 


i 


Very  little  spare  capacity,  but 
maintenance  of  effort  in  the  primary 
tasks  not  in  question 


Very  high  workload  with  almost  no 
spare  capacity.  Difficulty  in 
maintaining  level  of  effort 


Extremely  high  workload.  No  spare 
capacity.  Serious  doubts  as  to  ability 
to  maintain  level  of  effort 
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TSCWL1.  (con’t)  Place  the  workload  rating  in  the  blank  next  to  each  crewmember 
using  the  rating  scale  on  the  previous  page. 


Crewmembers 

Overall  Workload  Rating  For 
This  Mission 

Left  Seat 

Right  Seat 

If  you  assigned  a  workload  rating  of  ‘6’  or  higher  for  either  crewmember,  explain  why: 


TSCWL2.  Which  crewmember  was  the  ‘flying  pilot’  for  most  of  the  mission? 
Left  seat  _  Right  seat  _ 


TSCWL3.  What  percentage  of  the  time  was  the  crewmember  (left  seat  or  right  seat  in  question 
above)  the  ‘flying  pilot’  during  the  mission? 

_ % 


TSCWL4.  Rate  the  effectiveness  of  aircrew  coordination  as  defined  by  the  USAAVNC  Aircrew 
Coordination  ETP  and  TC  1-210. 

1  2  3  4  5 


Excellent  Good 


Average  Needs  Improvement  Unacceptable 
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Situation  Awareness 


Rating 

Check  one 

Crew  was  consistently  aware  of  all  entities  on  the  battlefield 

Crew  was  aware  of  the  battlefield  with  minor  or  insignificant  variation  between 

perception  and  reality. 

Crew  was  aware  of  the  battlefield.  Variation  between  reality  and  perception 
did  not  significantly  impact  mission  success. 

SA  needs  improvement.  Lack  of  SA  had  some  negative  effect  on  the  success 

of  the  mission. 

Lack  of  SA  caused  mission  failure. 

Describe  any  problems  that  aircrews  had  with  situation  awareness. 
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List  of  Symbols,  Abbreviations,  and  Acronyms 


3-D 

AAR 

AMRDEC 

AOI 

ARH 

ARL 

ATEC 

BHEAC 

BHIVE 

BWRS 

CAAS 

CAD 

CHEAC 

CPC 

EDS 

EUD 

FDT&E 

HFM 

HRED 

LEUE 

LUT 

PCI 

RACRS 

SA 

SART 

SIL 

SME 

SSQ 


3-Dimensional 
After- Action  Review 

Aviation  and  Missile  Research,  Development  and  Engineering  Center 
Area  of  Interest 

Armed  Reconnaissance  Helicopter 

Army  Research  Laboratory 

Army  Test  and  Evaluation  Command 

Blackhawk  Helicopter  Engineering  and  Analysis  Cockpit 

Battlefield  Highly  Immersive  Virtual  Environment 

Bedford  Workload  Rating  Scale 

Common  Aviation  Architecture  System 

Computer  Aided  Design 

Cargo  Helicopter  -  Engineering  Analysis  Cockpit 

Comanche  Portable  Cockpit 

Engineering  Development  Simulator 

Early  User  Demonstration 

Force  Development  Test  and  Evaluation 

Human  Figure  Modeling 

Human  Research  and  Engineering  Directorate 

Limited  Early  User  Evaluation 

Limited  User  Test 

Pilot-Crewstation  Interface 

Risk  and  Cost  Reduction  System 

Situation  Awareness 

Situation  Awareness  Rating  Technique 

System  Integration  Laboratory 

Subject-matter  Expert 

Simulator  Sickness  Questionnaire 
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TCM 

TRADOC  Capabilities  Manager 

TRADOC 

U.S.  Army  Training  and  Doctrine  Command 

TS 

Total  Severity 

UP 

Tactics,  Techniques,  and  Procedures 

UAS 

Unmanned  Aircraft  System 

VFR 

Visual  Flight  Rules 

WSRT 

Wilcoxon  Signed  Rank  Test 
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Intentionally  Left  Blank. 
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